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Character education is a fairiy new and rapidly evolving topic for 
curriculum interventions. It is also broad, often overlapping with 
other program areas. 

Character can encompass such moral and ethical values as 
respect, fairness, and caring— as well as responsibility, trust- 
worthiness, and citizenship. And it can refer to the demonstra- 
tion of these values in behavior, reasoning, and emotions. 

Some character education programs integrate activities in 
social studies, English, and math. Some create communities of 
students, teachers, and parents to foster respect and caring. 

And some use in-class stories, games, songs, and activity books 
to encourage the acceptance of other cultures. 

To set some bounds on this evolving topic, the WWC identi- 
fied programs that deliberately attempt to develop students’ 
character by teaching core values and that had most if not all 
of their lesson plans or prescribed activities directly related to 
instilling those values. Most programs did not enter our review 
because of this criterion. For programs that did, most had only 
one or two small studies that met our evidence standards. 

For these studies, student outcomes were measured in three 
domains: behavior, academic achievement, and knowledge, 
attitudes, and values. 

We looked at 93 studies of 41 programs that qualified for our 
review. Of these, 18 studies of 13 programs met our evidence 
standards, 7 without reservations, and 11 with reservations. (The 
identification of eligible programs ended in September 2005, and 
that of eligible studies, in July 2006.) 



In looking at the three outcome domains for the 13 programs: 

• Positive Action had positive effects on behavior O and on 
academic achievement Q 

• Too Good for Drugs and Violence had positive effects on 
knowledge, attitudes, and values Q 

• Too Good for Violence had potentially positive effects on 
behavior (if and on knowledge, attitudes, and values if 
Six other programs had potentially positive effects in one 

domain. Five had no discernible effects in any domain. 



Character education in practice 



Setting schoolwide principles. The School Improvement Com- 
mittee at Davis elementary, comprising the parents, teachers, 
administrators, and the school principal, supervises practices that 
promote the school’s values— such as cultural awareness and Inter- 
personal respect. Ms. Lopez, a fourth grade teacher, helps her class 
reflect on their decisionmaking and behavior, drawing on students’ 
personal experiences. After each lesson, the class agrees on rules 
of respectful and culturally sensitive behavior. Throughout the year, 
the students deepen their understanding of these concepts through 
class discussion, art projects, and schoolwide events. 



Character education in practice 



Integration into the curriculum. Teachers at Oaks high school 
work to infuse moral problem-solving and ethical thinking into the 
curriculum. In history and social studies, students explore the ethi- 
cal values found in the Declaration of Independence. In language 
arts, students write essays that focus on moral dilemmas. In math, 
students learn how to interpret statistical results from a survey about 
moral attitudes. In additional small-group discussions, facilitated by 
trained teachers and senior students, students talk about dilemmas 
in their personal life and how to resolve them. 
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Intervention Ratings for Character Education 

Each character education program that had at least one study meeting WWC 
standards (with or without reservations) received a rating of effectiveness in at 
least one outcome domain: behavior, knowledge, attitudes, and values, and 
academic achievement. The rating aims to characterize the existing evidence in a 
given domain, taking into account the quality of the research design, the statistical 
significance of the findings, the size of the difference between participants in the 



intervention and comparison conditions, and the consistency in findings across 
studies. 

The research evidence can be rated as positive, potentially positive, mixed, no 
discernible effects, potentially negative, or negative (see the WWC Intervention 
Rating Scheme) . Table 1 shows the effectiveness ratings for the 13 programs in 
the three domains (empty cells indicate that no evidence was reported). 



Table 1 Effectiveness ratings for 13 programs in three domains 




Note: Two programs had research findings in all three domains; five programs had research findings in two domains; and six programs had research findings in one domain. WWC intervention reports describe each program and provide 
information on the students, the cost, and the scope of use. To view the intervention reports piease click on the program name or go to www.whatworks.ed.aov . Foiiowing each program name is the deveioper’s website address. The 
research evaiuated addresses some but not aii grade ievels targeted by these interventions. Grade ieveis are reiated to student age and may affect outcomes. For a comparison of targeted grade ievels and grade ieveis in the studies 
reviewed by the WWC see Appendix A2. The rating for Building Decision Skills pertains to Building Decision Skills combined with Service Learning, 



O Positive effects: 
strong evidence of 
a positive effect 
with no overriding 
contrary evidence 




Potentialiy 
positive effects: 
evidence of a 
positive effect 
with no overriding 
contrary evidence 




Mixed effects: 
evidence of 
inconsistent effects 




No discernibie 
effects: no 
affirmative evidence 
of effects 




Potentialiy 
negative effects: 
evidence of a 
negative effect 
with no overriding 
contrary evidence 



O Negative effects: 
strong evidence of 
a negative effect 
with no overriding 
contrary evidence 
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Average improvement indices for each domain 

The WWC computes an average improvement index for each domain and each 
study as weii as a domain average improvement index across studies of the same 
intervention (see the Technicai Detaiis of WWC-Conducted Computations^ . 

The improvement index represents the difference between the percentile rank 
of the average student in the intervention condition and the percentile rank of 
the average student in the comparison condition. It can take on values between 
-50 and +50, with positive numbers denoting results favorable to the intervention 
group. Unlike the rating of effectiveness, which is based on four factors, the 
improvement index is based only on the size of the difference between the inter- 
vention and the comparison conditions.'' 

Behavior 



as aggressive or delinquent behavior). We reviewed outcomes in this domain for 
nine character education programs, and the average improvement index ranged 
from +0 to +19 percentile points (figure 1). 

Knowledge, attitudes, and values 

The knowledge, attitudes, and values domain includes student outcomes that 
center on moral and ethical reasoning, as well as attitudes, feelings, predisposi- 
tions, and self-perceptions that are associated with pro-social character (such as 
democratic values and empathy) or the lack of it (such racism, positive attitudes 
toward aggression). We reviewed outcomes in this domain for 10 character 
education programs, and the average improvement index ranged from +0 to +16 
percentile points across programs (figure 2). 



The behavior domain includes observed or reported student behavior outcomes 
that may be regarded as indicators of socially responsible character (such as 
participating in community service and supporting peers) or of the lack of it (such 



Academic achievement 

The academic achievement domain includes measures of academic achievement 
(such as standardized test scores) and measures of academic persistence or 



Figure 1 Behavior: average improvement 



Figure 2 Knowledge, attitudes, and values: average improvement 



Percentile points 

o 







Outcomes may include: 
Supportive, friendly, and helpful behavior 
Better ethical conduct^ 
Less fighting 





Positive Too Good Connect 

Action for Violence with 

Kids 



Too Good Lessons in Facing 
for Drugs™ Character History and 
Ourselves 



Caring Skills for Hearfwood 
School Adolescence Ethics 
Commu- Curriculum 

nity™ 




Too Good Too Good Building Caring Too Good Voices Lessons in Skills for Heartwood Facing 

for for Drugs™ Decision School for Drugs LACE Character Action Ethics History 

Violence Skills Commu- & Violence Program Curriculum and 

nity™ Ourselves 



1 . To enable comparisons across interventions, improvement indices are calculated from student-level findings only. In the case of Positive Action findings in the behavior and academic achieve- 
ment domains and in the case of Caring School Community findings in the knowledge, attitudes, and values domain, the average improvement index does not represent all the findings reviewed 
by the WWC, as some findings were reported on the classroom or school level, and student-level improvement indices could not be computed. For further details please see Technical Details of 
WWC-Conducted Computations . 

2. Ethical conduct includes honest, fair, rule-abiding, and respectful behavior toward children and adults during cooperative and competitive activities. 
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participation (such as attendance, retention, and graduation rates). We reviewed 
outcomes in this domain for three programs, and the average improvement index 
ranged from +0 to +16 percentile points (figure 3). 



Figure 3 Academic achievement: average improvement 
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Character education in practice 



Role models and service learning. The character education program at Lowell high 
school combines in-class activities with service learning. This year, the ninth-graders 
decided to contribute to the appearance of their downtown area. The students 
researched the type of plants most appropriate for this area, prepared a budget, and 
presented an action plan to the city council, which approved the plan. With the help 
of teachers and parents, they raised the money for the equipment and supplies. The 
students wrote about their experiences in language arts classes and discussed the 
moral and ethical values that they felt they developed through the service learning. 
Towards the end of the semester, the school principal met with student representatives 
to listen to their feedback about the character education and service learning efforts in 
the school. 



Character education in practice 



A comprehensive focus on caring. In Jefferson middle school, teachers and school 
administrators model caring behavior during class hours and school breaks and 
acknowledge students’ acts of kindness. Parents receive the school newsletter, which 
shares what the school does to promote a caring school community and invites families 
to take part in the school’s projects and special events. Teachers’ assessments of stu- 
dents’ caring behavior is included in report cards. A school climate survey, administered 
to students as part of the school’s self-assessment, has a scale on whether students 
experience faculty and their fellow students as being caring. 



Note: The character education practices present hypothetical scenarios based on elements 
recommended by the Office of Safe and Drug-Free Schools and the Character Education 
Partnership. 
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Table 2 Programs reviewed with no studies meeting WWC evidence screens 



A Celebration of Character (hUp://www.kkconsu!t.com/celebration.htm) 


Living Values: Educational Program (http://www.livingvalues.net/) 


AEGIS (Acquiring Ethicai Guideiines for individual Seif-governance; fiftp://www.aegis-character.com/) 


Love in a Big World (http://www.loveinabigworld.org/) 


Building Esteem in Students Today (BEST) (no website available) 


Making Meaning (http://www.devstu.org/) 


The Caring Habits Experience (http://www.jeffersoncenter.org/index.html) 


Open Circle (http://www.open-circte.org/main.asp?areaid=0) 


Changing Lives (http://www.mark1.org/changinglives/) 


Pathways to Character (http://www.epicforchildren.org/) 


Character Building & Reading Mastery (http://www.kkconsult.com/readitheedithtm) 


Peace Education (http://www.un.org/cyberschooibus/peace/index.asp) 


Character Counts! (http://www.charactercounts.org/) 


Project Wisdom (http://www.projectwisdom.com/) 


Character First! (www.characterfirst.com) 


Resolving Conflicts Creatively Program (http://www.esrnational.org) 


CHARACTERplus (http://characterpius.org/) 


Skills for Growing (http://www.lions-quest.org/) 


Community of Caring (http://communityofcaring.org/AboutCofC/overview.html) 


Smart and Good (http://www.cortland.edu/character/highschool/) 


ESSENTIAL Curriculum/ Project Essential (http://www.teelinstitute.org/home.htm) 


Success through Accepting Responsibility (S.T.A.R.; 
http://www. jeffersoncenter. org/jcenter/STAR. htm) 


The Giraffe Heroes Program (http://www.giraffe.org/) 


Tribes TLC® (http://www.tribes.com/) 


“1 Care” Parental Involvement Character Curriculum (http://www.icarenow.com/) 


Wisdom for Life (http://www.livewiremedia.com/wisdom.html) 


Learning for Life (http://www.learningforlife.org/lfl/) 


WiseSkills® (http://www. wiseskills.com/) 



Note: Following each program name is the developer’s website address. 

For more information about studies reviewed and WWC methodoiogy, piease see the WWC Character Education Technical Appendices . 
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Appendix 



Appendix A1 Extent of evidence 







Behavior 




Knowledge, 


attitudes, 


and values 


Academic achievement 


Intervention name 


Number of 
studies 


Sample size 
(schools/ 
students) 


Extent of 
evidence 


Number of 
studies 


Sample size 
(schools/ 
students) 


Extent of 
evidence 


Number of 
studies 


Sample size 
(schools/ 
students) 


Extent of 
evidence 


Building Decision Skills 


0 


0 


na 


1 


1/283 


Small 


0 


0 


na 


+ Service Learning 




















Caring School Community™ 


2 


16/2,336 


Moderate/large 


2 


10/2,303 


Moderate/large 


2 


16/3,000+ 


Moderate/large 


Connect with Kids 


1 


12/800 


Small 


0 


0 


na 


0 


0 


na 


Facing History and Ourselves 


1 


5/346 


Small 


1 


5/346 


Small 


0 


0 


na 


Heartwood Ethics Curriculum 


1 


4/870 


Small 


1 


4/858 


Small 


0 


0 


na 


Lessons in Character 


1 


7/141 


Small 


2 


22/433 


Moderate/large 


1 


7/141 


Small 


Positive Action 


2 


56/4,000+ 


Moderate/large 


0 


0 


na 


2 


56/4,000+ 


Moderate/large 


Skills for Action 


0 


0 


na 


1 


25/1,800 


Small 


0 


0 


na 


Skills for Adolescence 


1 


34/7,426 


Small 


0 


0 


na 


0 


0 


na 


Too Good for Drugs^"" 


1 


6/1,051 


Small 


2 


12/1,995 


Moderate/large 


0 


0 


na 


Too Good for Drugs 


0 


0 


na 


2 


27/504 


Moderate/large 


0 


0 


na 


& Violence 




















Too Good for Violence 


1 


10/999 


Small 


1 


10/999 


Small 


0 


0 


na 


Voices Literature and 


0 


0 


na 


1 


5/98 


Small 


0 


0 


na 


Character Education 





















na = not studied 

Note: All the programs that received a rating of moderate/large had at least four schools in each of the studies reviewed. A rating of “moderate to large” requires at least two studies and two schools 
across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is “small.” 
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Appendix A2 Targeted population 



Program name 


Targeted students (grades) 


Students in studies reviewed (grades) 


Building Decision Skills + Service Learning 


7-12 


12 


Caring School Community™ 


K-6 


K-6 


Connect with Kids 


3-12 


3-12 


Facing History and Ourselves 


6-12 


8 


Heartwood Ethics Curriculum 


K-6 


1-6 


Lessons in Character 


K-8 


4-5 


Positive Action 


K-12 


1-6 


Skills for Action 


9-12 


9-12 


Skills for Adolescence 


6-8 


6-8 


Too Good for Drugs™ 


K-8 


3, 4,6 


Too Good for Violence 


K-8 


3 


Too Good for Drugs and Violence 


9-12 


9-12 


Voices Literature and Character Education Program 


K-12 


6-7 



Note: This table presents a comparison of targeted grade ieveis and the grade ieveis in the studies reviewed by the WWC. Grade ieveis are reiated to student age and 
may affect outcomes due to differences in the students’ deveiopmental stages as weii as differences in schooi size and organization. 
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Appendix A3 Summary of statistically significant^ or substantively important^ positive findings 





Behavior^ 




Knowledge, attitudes, 


and values^ 


Academic achievement^ 




Statistically 




Statistically 




Statistically 


Academic 




significant 


Behavior across 


significant 


Knowledge 


significant 


achievement 




positive findings 


outcomes 


positive findings 


across outcomes 


positive findings 


across outcomes 


Building Decision Skills + Service Learning 












Leming 2001 


na 


na 


Ethical perspective 


ns, 


na 


na 


(quasi-experimental design) 








nsi 






Caring School Community 

San Ramon Study 


Spontaneous prosooial behavior 


ns. 


ns 


ns, 


ns 


ns. 


(randomized controlled trial with 


Supportive, friendly, & 


Substantively 




nsi 




nsi 


confounding problems) 


helpful behavior 


important 










The Six-District Study 


ns 


ns. 


ns 


ns, 


ns 


ns. 


(quasi-experimental design) 




nsi 




nsi 




nsi 


Connect with Kids 














Page & D’Agostino 2005 


Interpersonal behavior survey 


Statistically 


na 


na 


na 


na 


(quasi-experimental design) 




significant. 

Substantively 














important 










Facing History and Ourselves 

Schultz, Barr, & Selman 2001 


ns 


ns. 


ns 


ns. 


na 


na 


(quasi-experimental design) 




nsi 




nsi 






Heartwood Ethics Curriculum 














Leming 2000 


ns 


ns. 


ns 


ns. 


na 


na 


(quasi-experimental design) 




nsi 




nsi 







na = not studied (continued) 

ns = not statistically significant 
nsi = not substantively important 

1 . According to WWC criteria, if a program finds a statistically significant effect, then there is less than a 5% chance that this difference is due to chance. The level of statistical significance was calculated by the WWC and, where neces- 
sary, corrects for clustering within classrooms or schools, and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . For the formulas the WWC used to calculate the statistical 
significance, see the Technical Details of WWC-Conducted Computations . 

2. For rating purposes, the WWC considered the statistical significance of the findings and the magnitude of the effect, also called the effect size. An average effect size is the sum of all the effect sizes of the student outcomes in a study in 
a single domain divided by the number of those outcomes. The WWC considers an average effect size across all student outcomes in one study in a given domain to be substantively important if it is equal to or greater than 0.25. 

3. No studies showed statistically significant or substantively important negative findings in the domain. For a detailed description of the outcome measures, see Appendix A2 in the WWC intervention reports at www.whatworks.ed.aov . 
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Appendix A3 Summary of statistically significant^ or substantively important^ positive findings (continued) 





Behavior^ 




Knowledge, attitudes, 


and values^ 


Academic achievement^ 




Statistically 




Statistically 




Statistically 


Academic 




significant 


Behavior across 


significant 


Knowledge 


significant 


achievement 




positive findings 


outcomes 


positive findings 


across outcomes 


positive findings 


across outcomes 


Lessons in Character 














Dietsch, Bayha, & Zheng 2005 


ns 


ns. 


ns 


ns, 


Mathematics grades 


Statistically 


(randomized controlled trial) 




nsi 




nsi 


Attendance 


significant. 

Substantively 














important 


Lions Quest — Skills for Action 














Laird, Bradley, & Black 1998 


na 


na 


ns 


ns, 


na 


na 


(quasi-experimental design) 








nsi 






Lions Quest — Skills for Adolescence 












Eisen, Zellman, & Murray 2003 


Binge drinking 


ns. 


na 


na 


na 


na 


(randomized controlled trial) 




nsi 










Positive Action 














Flay et al. 2006 


Suspensions 


Statistically 


na 


na 


Grade retention 


Statistically 


(randomized controlled trial) 


Tobacco use 


significant. 








significant. 




Alcohol use 


Substantively 








Substantively 




Being drunk 
Illegal drug use 
Serious violence (boys) 


important 








important 


Flay & Allred 2003 


Violence rates 


Statistically 


na 


na 


The Florida Comprehensive 


Statistically 


(quasi-experimental design) 


Suspension rates 


significant. 






Aptitude Test (FCAT) 


significant. 






Substantively 








Substantively 






important 








important 


Too Good for Drugs™ 














Bacon 2000 


na 


na 


na 


ns, 


na 


na 


(randomized controlled trial) 








nsi 






Bacon 2003 


ns 


Statistically 


na 


ns, 


na 


na 


(randomized controlled trial) 




significant. 

Substantively 




nsi 










important 










na = not applicable 

ns = not statistically significant 

nsl = not substantively Important 












(continued) 
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Appendix A3 Summary of statistically significant^ or substantively important^ positive findings (continued) 





Behavior^ 




Knowledge, attitudes, 


and values^ 


Academic achievement^ 




Statistically 
significant 
positive findings 


Behavior across 
outcomes 


Statistically 
significant 
positive findings 


Knowledge 
across outcomes 


Statistically 
significant 
positive findings 


Academic 
achievement 
across outcomes 


Too Good for Drugs and Violence 

Bacon 2001a 

(randomized controlled trial) 


na 


na 


Perceptions of social and 
resistance skills 
Perceptions of emotional 
competence 


ns, 

Substantively 

important 


na 


na 


Bacon 2001b 

(quasi-experimental design) 


na 


na 


Perceptions of social and 
resistance skills 

Perceptions of emotional 
competence 

Positive attitudes towards 
non-violence 

Perceptions of assertiveness/ 
efficacy skills 


ns, 

Substantively 

important 


na 


na 


Too Good for Violence 














Hall & Bacon 2005 
(randomized controlled trial) 


Teacher checklist of student 
behaviors (20 week 
follow-up) — total score 


Statistically 

significant, 

Substantively 

important 


ns 


ns, 

Substantively 

important 


na 


na 


Voices Literature and Character Education 












Demetriades-Guyette 2002 
(quasi-experimental design) 


na 


na 


na 


ns, 

nsi 


na 


na 



na = not applicable 

ns = not statistically significant 

nsi = not substantiveiy important 
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Appendix A4 
Methodology 



WWC Topic Report 



Ninety-three studies provided data on 41 character education 
programs and were classified for the strength of their design. To 
be fully reviewed, a study had to be a randomized controlled trial 
or a quasl-experimental design. 

Eligibility for review 

Quasi-experiments eligible for review Include those equating 
through matching or statistical adjustment, regression disconti- 
nuity designs, and single-case designs. No studies based on the 
latter two types of designs were identified for the character edu- 
cation review. We are currently developing evidence standards 
for regression discontinuity designs and single-case designs. 

The review considered the properties of measurement instru- 
ments, the percentage of students, classrooms, or schools in the 
study sample that were not included in the reported results, and 
any sample characteristics or events that might serve as alterna- 
tive explanations for the observed effect. For details please 
see the WWC Evidence Standards . Long-term outcomes were 
preferred over immediate outcomes for inclusion in our analysis 
of program effects. 

The research evidence for programs that have at least one 
study meeting WWC evidence standards with or without reserva- 
tions is summarized in individual intervention reports posted on 
the WWC website. See http://www.whatworks.ed.aov . So far, 18 
studies of 13 character education programs have met evidence 
standards with or without reservations. The lack of evidence for 
the remaining programs does not mean that those programs 
are ineffective; some programs have not yet been studied using 
a study design that permits the WWC to draw any conclusions 
about their effectiveness. And for some studies, not enough data 
were reported (such as descriptive statistics of the findings) to 
enable us to confirm statistical findings. 

Rating of effectiveness 

Each character education program that had at least one study 
meeting WWC standards with or without reservations received 
a rating of effectiveness in at least one outcome domain. The 



Character Education 



rating of effectiveness aims to characterize the existing evidence 
base in a given domain. The intervention’s effects based on the 
research evidence can be rated as positive, potentially positive, 
mixed, no discernible effects, potentially negative, or negative. 

The rating of effectiveness takes into account four factors: the 
quality of the research design; the statistical significance of the 
findings; the size of the difference between participants in the 
intervention and the comparison conditions; and the consistency 
in findings across studies (see the WWC Intervention Rating 
Scheme) . 

The level of statistical significance was reported by the study 
authors or, where necessary, calculated by the WWC to correct 
for clustering within classrooms or schools and for multiple com- 
parisons. Because of these corrections, the level of statistical 
significance as calculated by the WWC may differ from the one 
originally reported by the study authors. For an explanation, see 
the WWC Tutorial on Mismatch . For the formulas that we used to 
calculate statistical significance, see Technical Details of WWC- 
Conducted Computations . If the average effect size across all 
outcomes in one study in a single domain is at least 0.25, it is 
considered substantively important, contributing toward the 
rating of effectiveness. See the technical appendices of the 
character education intervention report for further details. 

Extent of evidence 

The evidence base rating represents the size and number of 
independent samples that were assessed for the purposes of 
analysis of the program effects. A “moderate/large” evidence 
base requires at least two studies and two schools across stud- 
ies within one domain, and a total sample size across studies of 
at least 350 students or 14 classrooms. Otherwise, the evidence 
base is considered to be “small.” The WWC is currently working 
to define a “large” evidence base. This term should not be con- 
fused with external validity, as other facets of external validity, 
such as variations in settings, important sub-groups of students, 
implementation, and outcomes measures, were not taken into 
account for the purposes of this rating. 
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Appendix A4 
Methodology 

(continued) 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each domain and 
each study as well as a domain average improvement index 
across studies of the same intervention (see the Technical 
Details of WWC-Conducted Computations) . The improvement 
index represents the difference between the percentile rank of 
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the average student in the intervention condition and the percen- 
tile rank of the average student in the comparison condition. The 
improvement index can take on values between -50 and +50, 
with positive numbers denoting results favorable to the interven- 
tion group. Unlike the rating of effectiveness, the improvement 
index is based only on the size of the difference between the 
intervention and the comparison conditions. 
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1. Complete data are not reported: the WWC could not compute effect sizes based on the data reported. 

2. Does not use a strong causal design: this study, which used a quasi-experimental design, did not establish that the comparison group was equivalent to 
the intervention group at baseline. 

3. Does not use a strong causal design: this study did not use a comparison group. 

4. Does not use a strong causal design: this study administered the initial measurement (pretest) after the intervention began. 

5. Does not use a strong causal design: this study did not use a quantitative design to assess student outcomes. 

6. The outcome measures are not shown to be valid or reliable. 

7. The outcome measures are not relevant to this review: The parameters for this WWC review specified student outcome measures but this study did not 
focus on students. 

8. Does not use a strong causal design: there was only one intervention and one comparison unit, so the analysis could not separate the effects of the 
intervention from other factors. 

9. The sample is not appropriate to this review 

10. The outcome measures are not relevant to this review: The parameters for this WWC review specified student outcome measures but this study did not 
focus on students. 

11. Does not use a strong causal design: this paper is a review and summary of other research. 

12. Does not use a strong causal design: this study used a qualitative approach to report findings. 
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